Your phone buzzes and, before you decide anything, your hand is already moving. A song from your school years comes on and you feel something that has nothing to do with the room you are in. You pull the lever on a vending machine that ate your coins last week, and press it again, hard, as if that would help.
None of these needed a thought. They are the footprints of learning that happens without anyone teaching you: a signal that now predicts something, an action that paid off, a reward that came a little less or a little more than expected. Psychologists once thought of this as a rat-and-pigeon topic. It turned out to be the machinery under advertising, games, addiction, parenting, classroom management, and even the training of artificial agents.
A graduate of this course keeps one idea above the rest: learning is driven by surprise. When the world matches your prediction, nothing is learned. When it doesn't, the size of the surprise decides how much you change.
Ivan Pavlov was studying digestion in dogs (work that won him the 1904 Nobel Prize in Physiology or Medicine) when he noticed that the dogs began to salivate before the food arrived, at the sight of the keeper or the sound of his steps. He turned the nuisance into the experiment. A neutral signal, such as a bell, repeatedly came just before food. The food (the unconditioned stimulus) already caused salivation. After enough pairings, the bell (the conditioned stimulus) caused it too.
The modern reading is less "association by closeness" and more "the bell became a forecast". The signal has to be informative: if food arrives just as often without the bell, the bell stays meaningless, however often they co-occur (Robert Rescorla, 1968). Biology also sets limits. In a famous 1966 experiment, John Garcia and Robert Koelling found that rats readily link a taste with later sickness, even hours later, but do not as readily link a flashing light and noise with sickness. It is why one bad oyster can end a love of oysters for years.
Humans do the same. A brand jingle that predicted a good time, a sound linked to a bad dentist visit, and the sinking feeling at a certain email notification are all classical conditioning. The evidence for fear learning in humans is not the 1920 "Little Albert" demonstration, which had one baby, no proper controls and an unsettled aftermath: researchers still dispute the child's identity and health. Later laboratory work established the effect properly, with consent, and with ways to undo it.
That undoing is extinction: present the bell without the food and the response fades. The old association is not erased; it is overlaid by a new, competing one, which is why fear can return after a break and why exposure-based therapies work by repeating safe encounters. (A hint of Clinical Psychology, 35/27.)
Classical: a signal comes to predict an outcome that happens anyway. Operant: an action is followed by a consequence that changes how often it recurs. Tap your answer.
Edward Thorndike (1898) put cats in puzzle boxes and watched them stumble onto the latch, then do it faster each time. He called it the law of effect: an action followed by a satisfying result becomes more likely. B. F. Skinner (The Behavior of Organisms, 1938) built on it, with a box in which an animal's lever pressing was recorded and rewarded under rules the experimenter set.
The vocabulary is a trap, so here it is in one table. "Positive" and "negative" mean added or removed. "Reinforcement" and "punishment" mean the behaviour becomes more or less likely.
| Behaviour more likely | Behaviour less likely | |
|---|---|---|
| Something added | Positive reinforcement: praise after tidying | Positive punishment: a scolding |
| Something removed | Negative reinforcement: the alarm stops when you get up | Negative punishment: losing screen time |
Two things graduates know. Negative reinforcement is not punishment; it is escape that rewards. And the effects of punishment are weaker and less reliable than people expect: it suppresses behaviour while it is applied and teaches what to avoid, not what to do. Teaching a new behaviour with a reward works better than ending an old one with a penalty, and animal trainers such as the ones in Karen Pryor's books build whole careers on that.
Not every response is rewarded in daily life, and Skinner and Charles Ferster (1957, Schedules of Reinforcement) found that the pattern of rewards matters as much as the amount. The four classic rules: a fixed ratio (every fifth press), a variable ratio (on average every fifth, but unpredictable), a fixed interval (the first press after ten seconds) and a variable interval (the first press after a random wait around ten seconds). In animal records the typical shapes are: fixed ratio gives bursts and a pause after each reward; variable ratio gives a fast, steady rate with almost no pause; fixed interval gives a scallop, slow after a reward and speeding up as the time nears; variable interval gives a steady moderate rate.
Try the rules yourself. This is a person tapping, not a pigeon, and you know what is happening, so your pattern will differ from an animal's. What the graph does honestly show is how often the rule paid out.
Reward = a tick on the line. Variable ratio draws each next requirement uniformly from 1 to 9 (mean 5); variable interval draws each wait uniformly from 2 to 18 s (mean 10). "No more rewards" is extinction.
Slot machines and many app notifications work as variable-ratio schedules, which is one reason they hold behaviour so tightly. The honest limits: that is an analogy from animal behaviour, not proof that any one product causes addiction, and the size of the effect in people is debated. Another fact graduates know is the partial reinforcement extinction effect: behaviour that was rewarded only sometimes lasts longer once rewards stop than behaviour rewarded every time, because "no reward yet" was always part of the deal.
Why does a bell stop teaching once it predicts the food? Leon Kamin showed in 1969 that if an animal has already learned that light predicts shock, then a tone added to the light is barely learned, though it is paired with the shock just as often. This is blocking. The tone adds no surprise, so nothing is left to learn.
In 1972 Robert Rescorla and Allan Wagner wrote the idea down. In words: each trial, the change in how strongly a cue predicts the outcome is proportional to the error, the difference between what happened and what all the present cues together predicted.
A worked number: with a learning rate of 0.3 and an outcome of 1, a fresh cue A gains 0.3 on trial one, then 0.3 × 0.7 = 0.21 on trial two, and so on, leaving it near 0.99 after ten trials. Now add cue B alongside A, which already predicts 0.99. The error is only about 0.01, so B gets only about 0.003. Compare B learned beside a fresh A: they share the prediction and each reach 0.5.
Blocking: 10 trials of A, then 10 of A+B. Control: 20 trials of A+B. Extinction: 15 trials of A with reward, then 15 without. Outcome strength λ = 1 when rewarded, 0 when not.
The model is famous because it is simple and because it was wrong in useful ways. It cannot explain every effect (latent inhibition, for example), and later models added attention and timing. But the big idea survived and moved: learning follows the gap between expectation and outcome.
In the 1990s Wolfram Schultz recorded single dopamine neurons in monkeys. Early in training they fired when an unexpected juice reward arrived. After training, they fired at the cue that predicted the juice and no longer at the juice itself. If the promised juice failed to arrive, their firing dipped below baseline at the moment it was due. Schultz, Dayan and Montague (1997) showed that this matches a reward prediction error, the quantity in temporal-difference learning, an extension of Rescorla–Wagner to time.
The toy below uses that model. Each of six steps after the cue carries a value; the signal is how far each step's reward-plus-next-value differs from what was expected. Train it, then look at where the signal sits.
Temporal-difference model: learning rate 0.3, discount 0.95, reward 1 after the sixth step. Bars above the line are positive error, below are negative. This is a model of the signal, not a recording.
At zero trials, the whole error sits at the reward. After enough trials it has moved to the cue (about 0.74 in this setting, which is the discount to the power six) and the reward produces almost none. Omit the reward after training and the error at that moment is strongly negative. This is why dopamine is better described as a signal for learning and wanting than as "the pleasure chemical", the point of Dopamine Is Not the Pleasure Chemical (01/08). It is a strong, influential account, and not the only one: researchers also link dopamine to motivation, effort and salience, and the field argues over how much of its activity a single prediction-error idea explains.
With enough repetition in a stable setting, an action stops being a choice made for its result and becomes a response triggered by the setting. Rodent experiments (Adams and Dickinson, 1981) show it when a reward is made unpleasant and the animal keeps pressing anyway. How cleanly that split between goal-directed and habitual control carries over to people is debated.
The better-known number is how long it takes. In a 2010 study, Phillippa Lally and colleagues had volunteers choose a simple daily behaviour and rate how automatic it felt over twelve weeks. The time to reach a plateau of automaticity averaged 66 days, ranging from 18 to 254, and missing one day did not noticeably hurt. "It takes 21 days" has no such basis; it traces to a self-help book, Maxwell Maltz's Psycho-Cybernetics (1960), as commonly told.
Model: automaticity = 1 − e−3t/D, where D is the days to reach 95% of the plateau (set to 18, 66 or 254 to match the range Lally et al. reported). The shape follows their curve; your numbers will differ with the behaviour and the person.
The practical lesson is the cue. A habit attaches to a setting, so the same action in a new place is a new start, and the easiest habits are the ones with a clear, repeated trigger.
People and many animals learn by watching. In the Bobo doll studies (Bandura, Ross and Ross, 1961), nursery-age children who saw an adult hit an inflatable doll were more likely to hit it themselves than those who saw a calm adult. Bandura's later work stressed that watching someone get rewarded or punished matters (vicarious reinforcement), and that beliefs about one's own ability, which he called self-efficacy, shape what people try.
The honest caveats: a doll is not a person, children may guess what the experimenter wants, and the studies show imitation in a lab, not that screens cause real-world violence, which later research addresses with mixed and debated results. What holds up is the basic claim that modelling shapes behaviour.
Behaviour is shaped by what predicts what, and by how much the world differs from the forecast. Signals learn to predict, actions are strengthened by their consequences, the timing of rewards sets how stubborn a behaviour is, and dopamine neurons carry the surprise that drives it all. Habits are what a repeated cue leaves behind, and they take months, not weeks.
Try this: this week, notice one cue-and-response pair of your own (the notification and the reach for the phone, the smell and the craving). Write down the cue, the action and the reward. Then change just the cue and see what follows.
Sources. Pavlov (Nobel Prize in Physiology or Medicine 1904 for work on digestion); Rescorla 1968, Journal of Comparative and Physiological Psychology 66 (contingency); Garcia & Koelling 1966, Psychonomic Science 4; Watson & Rayner 1920 and later critiques (the child's identity and health are disputed; see Wikipedia's summary of Beck, Levinson & Irons 2009 and Digdon et al. 2014 for the competing identifications); Thorndike 1898; Skinner 1938; Ferster & Skinner 1957; Kamin 1969; Rescorla & Wagner 1972; Schultz, Dayan & Montague 1997, Science 275; Adams & Dickinson 1981; Lally et al. 2010, European Journal of Social Psychology 40 (mean 66 days to plateau, range 18 to 254); Maltz 1960 as the usual source of "21 days"; Bandura, Ross & Ross 1961. Schedule shapes are the textbook animal findings. Every number an interactive shows was computed in node: R–W at rate 0.3: A after 20 trials 0.999; blocking leaves B at 0.014 after ten compound trials; control A and B 0.50 each; extinction A 0.005 after 15 unrewarded trials. TD model: cue error 0.735 (0.956) after 50 or more trials; omitted reward, error −1.00. Habit model at day 21: 97% (D = 18), 62% (D = 66), 22% (D = 254).
This describes research, not medical advice. If you are struggling, a doctor or local service can help; findahelpline.com lists free helplines by country.